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ABSTRACT 



Methods and apparatus are provided for driving a vehi- 
cle from a remote control station achieving tele-opera- 
tion of the vehicle. A television sensor mounted on the 
vehicle provides video image information which is re- 
duced in bandwidth by a factor of approximately 1000:1 
for transmission by narrow band RF data link to a re- 
mote control station. The large bandwidth reduction is 
accomplished by first sampling the video output of the 
sensor at a reduced frame rate and then compressing the 
data further by standard data compression, techniques. 
Vehicle position and attitude data which may be de- 
rived from the vehicle's on board inertial reference unit 
are also transmitted via narrow band data link to the 
control station. At the control station, the data is first 
reconstructed by a technique which is the complement 
of the compression technique, and the instantaneous 
position and attitude data are used to compute trans- 
form coefficients that are used by a pipeline processor 
to extrapolate the frame data to generate a real time 
video display that enables an operator to drive the vehi- 
cle by transmitting appropriate control signals to the 
vehicle. 

11 Claims, 5 Drawing Sheets 
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as high resolution data only that portion corresponding 

HUMAN ENGINEERED REMOTE DRIVING to the eye position of the operator as he views the dis- 

SYSTEM play, the remainder of the image being transmitted from 

the vehicle as low resolution data constructed from the 

CROSS REFERENCE TO RELATED 5 high resolution memory. In a second mode of operation, 

APPLICATION the occulometer control is removed and all of the data 

The subject matter of this application is related to in memory is read out at a slower transmission rate and 

copending application Ser. No. 06/788,652 filed Oct. displayed as a fixed image at the control station. 

14, 1985, by Karl M. Fant and assigned to the assignee Kanaly makes no attempt to either reduce the frame 

of this application. The disclosure of application Ser. 10 rate itself or compensate for a reduced bandwidth af- 

No. 06/788,652 is incorporated herein by reference. forded by a reduction in the frame rate. The Kanaly 

RACKOROT JNn OF THE INVENTION approach to data compression is incapable of providing 

BACKGROUND OF THE INVENTION mQre thjm a 5Q:1 compression over conventional video 

Field of the Invention without severely degrading the imagery presented to 

The present invention generally relates to remotely 15 the operator, and falls far short of the 1000:1 compres- 

controlled vehicles and, more particularly, to a system sion afforded by the present invention, 

for transmitting video and motion data via a narrow Graham performs image compression based on a 

band radio frequency (RF) data link for interactive ranging laser sensor which measures the range rate to 

tele-operation of robotic vehicles. _ the objects being imaged. The Graham system performs 

20 image compression by sub-sampling a frame (or an 
Description of the Prior Art field), thereby blurring it. It adaptively recon- 
Remotely controlled vehicle systems using video structs the image by superimposing the blurred frames 
information transmitted from a vehicle to a control to achieve a less blurred image if the sensor is not mov- 
station for tele-operation of the vehicle are described in ing rapidly. Thus, it is only capable of approaching the 
U.S. Pat. Nos. 3,557,304 and 3,564,134 to Rue et al. high resolution in the reconstructed image when the 
4,405,943 to Kanaly and 4,682,225 to Graham. Typi- sensor itself is relatively stationary as in the case of 
cally, such systems involve the control of a remote space applications for which it was intended. In high 
vehicle by an operator at a control station to which velocity applications like aerial and ground vehicles, 
information regarding the vehicle's position, attitude this technique would result in continuously blurred 
and other operational parameters is displayed in real imagery because the scene motion precludes the super- 
time so that the operator is able to respond with appro- imposition of several sub-sampled frames. Moreover, no 
priate control signals to drive the vehicle. One or more reduction of the frame rate itself is proposed (nor the 
television sensors mounted on the vehicle provides compensation thereof), and this limits the Graham sys- 
image data depicting the scene, that is, the operating tem t0 not more 32: i compression, and even that 
environment of the remote vehicle, which data is trans- wou i<i result in a severely degraded image in the case of 
mitted by a communication link to the control station rap idly moving sensors. In fact, the resultant image 
where the scene is displayed for the operator's refer- WQuld be s0 b \ me & t h at j t wou id be equivalent to a 
ence. Control signals are returned to the vehicle by the « m y 0 pia" of 20/700 compared with full bandwidth 
operator providing interactive control of the vehicle. imagery 

The control signals may require a separate link, as may 40 Qther ' . M tents describmg remotely con- 

the transmission of information regarding the operation vdddca include US . Pat Nos . 4,018,405 to 

of the remote vehicle. fiak 4 096 380 to Eichweber, and 4,636, 1 37 to Lemel- 

In the remote control sys terns described in the pa tents sQn None of ^ g describcd in these patents 

to Rue et al., a wide angle lens is employed m a televi- m q{ compression , The patent to 

sum camera mounted in the cockpit of the controUed £ ^ ^ ^ , / indudi a mis . 

vehicle and a hemispherical viewuig screen is provided * tnmsmits 

at the ground station. Additionally, in he later patent to P > station 

Rue et al. mentioned above, a second television camera 77* *u j- i a ^ „ * ^^^^ 

having a zoom lens is placed in the nose of the remotely where they are displayed for an operator who generates 

controlled vehicle. No compression of the video infor- 50 control signals as necessary for transmission via the 

mation is performed in either one of the Rue et al. sys- laser beam b f* * the ve ^ le *> r S»! dance - ^chweber 

temg r also uses a laser beam, but of particular significance 

As is discussed in the patent to Kanaly, mentioned with respect to the invention disclosed herein is the 

above, the transmission of high resolution video data discussion of the disadvantage of the use of optical 

from a television type sensor requires a wide band data 55 fibers. Lemelson describes a different kind of robotic 

link. Kanaly describes prior art systems in which occu- mechanism employing a television camera and short 

lometer equipment is employed to limit the portion of wave transmitters. 

the scene being displayed as a high resolution image to As noted above in connection with the discussion of 
that actually being observed by the operator. The occu- the Kanaly patent, it will be further apparent from the 
lometer senses the portion of the displayed scene being 60 description of the present invention that the transmis- 
viewed and generates signals which are used on board sion of video data via a narrow band RF data link and 
the remote vehicle to limit the high resolution imaging the displaying of the data at the control station to pro- 
of a scene to that being observed. The Kanaly patent vide the instantaneous information required for the 
itself discloses a system in which bandwidth is reduced operator to close the control loop is an important aspect 
without this control of the camera aboard the vehicle 65 of this invention. In this regard, U.S. Pat. No. 3,962,537 
by imaging the entire field of the camera as high resolu- to Kearns et al. discloses a reconnaissance system which 
tion data and digitizing and storing it on the vehicle. includes a video RF link for transmitting optical image 
Occulometer equipment is used to extract from memory data from a camera carried in a gun launched projectile 
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to a remote location. U.S. Pat. No. 4,661,849 to Hinman by placing individual, normally detailed, objects with 

describes the transmission of a sequence of television high fidelity (CSI) on a specified surface or background 

camera images over a bandwidth-limited channel. The which may be CIG or CSI generated. A CGSI scene is 

method employs a motion estimation process for estima- constructed much in the manner of a CIG scene with 

tion, for successive image frames, a measure of motion 5 the surface elevations and objects locations laid out in a 

displacement between the images in the sequence. A uniform grid. The individual objects used in the scene 

navigation system is described in U.S. Pat, No. are tailored for perspective, location and transformation 

4,688,092 to Kamel et al. in which images from a satel- including size position, rotation, warp and intensity are 

lite camera are transmitted to an operations center on performed on each image as required. Like CIG sys- 

the ground where information is derived for orbit deter- 10 terns, the principal application for CGSI technology has 

mination and satellite control. been m the are of visual training simulators wherein a 

As discussed in the Hinman patent, a number of band- display scene is composed and constructed from a li- 

width compression techniques are available for various ^rary of images with sufficient processing speed to 

applications, among them teleconferencing; however, pennit rea j time ( 30 Hz) presentation to the observer, 

such systems take advantage of the fact that major ele- 15 Further information on the CGSI technique may be had 

ments of a scene are not changing and transmit only witn re f er ence to U.S. Pat. No. 4,667,190 to Karl M. 

those portions that have changed. Teleconferencing Fant and the aDO ve-referenced copending application 

utilizes a stationary camera, while for remote control of Ser No 06/788,652 to Karl M. Fant 
a robotic vehicle, the camera is mounted on a moving 

vehicle. 20 SUMMARY OF THE INVENTION 

The present invention makes use of computer image u is therefore a pr i nc i pa i obje ct of the present inven- 

generation (CIG) and computer synthesized imagery tion tQ oyide a tem for the mteractivc , rea i ^ 

(CSI) techniques for reconstructing a real time video tele ^ peration of a robotic vehic ie at high speeds, 

image by warpmga^o d^ u is anQther Qbject of ^ invent ^ on t0 ide a 

pie application area for CIG in the past has been that of 25 ^^^^ link between a robotic vehicle md a 

visual traimng simulators which present scenes to an remote Qr which is hardened tQ electronic 

observer or trainee to allow the observer to practice counter 

some task, such as flying an airplane. In a flight Simula- T . . n „™ • ^ *r 
tor, a three-dimensional model of the desired "gaming , li » a furth f an « m ° r s ?f Clflc f of the lnven - 
area'Ms prepared and stored on magnetic disk or * inula? 30 tl0 ° t0 P'™ de a ^ghly optimized data compression 
bulk storage media. This model is called the visual data svstem wm< * makes P osslble nar f ° w ban *vidth televi- 
base.The visual simulator combines an image generator slon data transmission while at the same time 
with an electro-optical display system such as a cathode Priding a high resolution video display of instanta- 
ray tube (CRT) or similar display. The image generator P osltlon mfonnatton for an operator of a robotic 
reads4» blocks-of three-dimensional data from the disk 35 vem ^te. t 
and transforms this data into two-dimensional scene It is yet another object of the mvention to provide a 
descriptions. The two-dimensional data are converted technique for performing a large ratio compression in 
to analog video that is presented to the operator or excess of 1000:1 of video imagery to enable tele-opera- 
trainee via the display. The generated imagery is meant tion of vehicles over a narrowband data link 
to be representative of the true scenes that the operator 40 The tele-operation or remote control of robotic vehi- 
would see if the operator were actually performing the cles » ™ contrasted with the autonomous operation of 
task being simulated. The generation of the display such vehicles, requires a communications link that in- 
images is said to be in "real time" which is normally eludes the operator as well as the vehicle. For such 
taken to mean 30 frames per second, as in the U.S. tele- operation, a real time image of the scene of the vehicle 
vision standard. In order to achieve a real time display, 45 operating environment as viewed by a sensor on the 
pipelined processors are conventionally used in CIG vehicle must be displayed to the operator, along with 
systems. CIG systems are described in detail in the book otner vehicle operational data, and vehicle control sig- 
entitled Computer Image Generation edited by Bruce J. nals m ^ be transmitted to the vehicle. In addition, the 
Schacter and published by Wiley-Interscience (1983). communications or data link should be non-line-of-sight 
CSI technology also generates images such as, for 50 (NLOS) and, in the case of a hostile environment such 
example, video displayable images from a data base, but as a battlefield, secure and hardened against electronic 
the objects and surfaces stored in the data base are rep- counter measures (ECM). 

resented as real-world electromagnetic media images of The real time display presented to the operator re- 
objects and surfaces rather than mathematical models quires instantaneous feedback of vehicle position and 
thereof as in CIG. Thus, whereas CIG uses a computer 55 attitude from a television camera mounted on the vehi- 
to generate imagery from a purely mathematical data cle. The transmission of this data in digital form up to 64 
base, CSI uses a computer to insert objects into a scene million bits per second requires a wide band data link, 
based on stored real-world images. Although CIG pro- Fiber optic data links offer sufficient bandwidth and are 
vides excellent control of a scene to be constructed and ECM hardened, but as suggested in Eichweber patent 
displayed for interaction in an environment, the fidelity 60 mentioned above, other problems are encountered in- 
is low and thus realism in the displayed scene is poor. eluding a lack of mechanical ruggedness, payout and 
CSI is just the opposite; that is, fidelity is excellent, but retrieval, battlefield repairs and cost, 
the control. over scene construction is restricted. Narrow band RF data links with a capacity of less 
The technique employed by the invention is a merger than 64 thousand bits per second are available in the 
of CIG and CSI technologies to form "Computer Gen- 65 battlefield environment and are NLOS secure and ECM 
erated Synthesized Imagery" (CGSI) as described in hardened. However, conventional bandwidth compres- 
U.S, Pat. No. 4,645,459 to Carl P. Graf et al. As de- sion techniques noted above fall far short of the approx- 
scribed in the Graf et al. patent, a scene is constructed imately 1000:1 ratio required to compress the video 
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bandwidth down to the bandwidth achievable in nar- with instantaneous position and attitude data. A pipeline 
row band RF data links commonly used for telemetry. controller oversees the transformation of snapshot data 
According to the present invention, this problem is and the input of the next snapshot into the pipeline. The 
solved by the provision of a bandwidth compression pipeline controller provides the interface which allows 
system which reduces the video data from a television 5 for a real time presentation of a continuous video dis- 
seisor to a narrow bandwidth which can be accommo- play generated from a series of snapshots. To drive the 
dated by an RF data link, thereby closing the loop vehicle, commands are executed by the operator result- 
around the driver of a tele-operated vehicle. Briefly ing in changes in vehicle position and attitude. The 
described, "snapshots" of the vehicle's operating envi- changes are transmitted back to the control station 
ronment are obtained by sampling on board the vehicle. 10 where they are included in the display to continuously 
The video data from the television camera is therefore update the latter. 

at a reduced frame rate, and this data is subjected to a In an alternative form of the invention, a single frame 

further bandwidth compression using conventional image transmitted from the vehicle is presented in fro- 

techniques for transmission on an RF data link to the zen frame form. Information as to position and attitude 

control station. Instantaneous vehicle position and atti- IS of the vehicle when the picture was taken and current 

tude data are also derived from an inertial reference unit position and attitude are used to construct a dynamic 

on the vehicle and are digitized for transmission over graphics overlay corresponding to shape and position of 

the same or another RF data link to the control station. the vehicle is superimposed on the frozen frame pres- 

At the control station, a real time image transformation ented to the operator. In this display, the vehicle form 

device continuously transforms the last received frame 20 appears to move into the still frame, representing instan- 

information using the instantaneous position and atti* taneous vehicle position which the operator controls, 

tude information to account for actual vehicle motion. The next still frame received at the control station re- 

The transformed video image is displayed for the opera- places the old frame and the current position and atti- 

tor and has the appearance of continuous video, with tude of the vehicle relative to the new frame is included 

the received freeze-frame snapshot being extrapolated 25 in the display. 

using the instantaneous vehicle position and attitude DESCRIPTION OFTHF DRAWINGS 
information until the next frame comes along. This BRIEF DESCRIPTION OF THE DRAWINGS 
allows the operator to interact by using the controls at The foregoing and other objects, aspects and advan- 
his station to generate signals to drive the vehicle. tages of the invention will be better understood from 
An important basic feature of the invention is the 30 the following detailed description of a preferred em- 
large reduction in bandwidth achieved. A reduction bodiment of the invention with reference to the draw- 
ratio in excess of 1000:1 is achieved by sampling the ings, in which: 

video at a frame rate of one frame every one to three FIG. 1 is an illustration, largely conceptional and 

seconds in contrast to the thirty frames per second stan- functional, showing the basic components of the remote 

dard-tor U.S. television and further subjecting the frame 35 drive system of the invention; 

data to conventional bandwidth compression tech- FIG. 2, is a block diagram showing the components 

niques providing a nominal additional 10: 1 to SO: 1 com- of the system which are mounted on the remote vehicle; 

pression ratio. There exists a large number of well FIG. 3 is a block diagram of the system components 

known techniques for bandwidth compression that rely which are located at the control station; 

on (1) time based recursive estimation, (2) transform 40 FIG. 4 is general functional block diagram of the 

based estimation of scene content such as Hadamard, pipeline warp processor which is part of the system 

Haar and Discrete Cosine Transform, (3) variable reso- shown in FIG. 3; and 

lution based on where the "eye" is looking, as in the FIG. 5 is a pictorial illustration of the warping pro- 
Kanaly patent, or (4) a hybrid technique based on a cess performed by the pipeline warp processor, 
combination of the foregoing. Individually, these 45 nP TATT Fn nF^mrimnN op a PRPFFRRPn 
achieve compression ratios of up to 20:1 or at most 50:1 DE ™nS^^^ 
which is not sufficient for the purpose of this invention, EMBODIMENT OF THE INVENTION 
although thy can be used in processing the data from Referring now to the drawings, and more particu- 
sampled frames. This invention exploits the interframe larly to FIG. 1, the overall remote drive system is 
data from the vehicle's inertial reference unit giving 50 shown as consisting of two major components, namely, 
instantaneous position and attitude data to continuously a vehicle and vehicle based equipment enclosed in 
transform the last received frame and provide a real dashed lines 10 and a control station enclosed in dashed 
time image of actual vehicle motion. lines 12. A number of narrow band RF links identified 
Another significant feature of the invention is the real by numerals 14, 16 and 18 are shown which provide 
time image transformation that is accomplished at the 55 communications between the vehicle based equipment 
control station, after the data including scene and in- 10 and the control station 12. Three links are shown, 
stantaneous vehicle position and attitude information is one for each of the basic types of data being communi- 
received via narrow band data link, which provides a cated between the components 10 and 12; namely, corn- 
continuous video type display including any perspec- pressed bandwidth video data over link 14, instanta- 
tive changes. At the control station, the compressed 60 neous vehicle position and attitude data over link 16, 
data is first reconstructed by a technique complement- and vehicle control data over link 18. Obviously, one 
ing that used on board the vehicle. Position and attitude narrow band link might suffice, or additional links could 
data from the vehicle's on board navigation system are be used, if necessary. An operator 20 is shown whose 
sent continuously to the narrow band receiver. A pipe- task it is to interact with the data presented at the con- 
line data processor performs the "warping" transforma- 65 trol station to drive a remote vehicle 22. 
tion that provides real time display at the remote con- Vehicle 22 can be of any type; i.e., capable of moving 
trol station. Image elements of the sampled frames, i.e., on land or sea or in the air. For the sake of this descrip- 
snapshots, are manipulated or "warped" in accordance tion, the vehicle 22 is depicted as having wheels and 
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therefore may be a land-based vehicle capable of tra- of continuous video by extrapolating between snapshots 
versing various types of terrain under the control of using vehicle, i.e., camera, position and attitude infor- 
operator 20. mation, 

A television camera 30 is mounted on vehicle 22 at FIGS. 2 and 3 show the vehicle on board equipment 
the highest possible point on the vehicle, taking into 5 and the control station, respectively, in more detail. In 
account clearance limitations. A charge coupled device FIG. 2, television camera 30 is shown mounted on a pan 
(CCD) camera with fast electronic shuttering (1/100 to and tilt mechanism 50 for motion independent of the 
1/2000 seconds) may be required to avoid blurring. If vehicle 22 platform. The latter may receive commands 
the camera 30 is mounted on a pan and tilt mechanism, from the operator 20 at control station 12. Camera 
pickofTs 32 are required to reference the camera 30 to an 10 pointing pickoffs 32 provide signals regarding camera 
intertial reference unit 34 on the vehicle to provide 30 position and attitude to a data formatter/processor 
exact camera position. The inertial reference unit 34 52 which also accepts position and attitude information 
may be part of the internal control system on vehicle 22 from inertial reference unit 34 on board vehicle 22. 
or it may be required especially for this application. In Vehicle position and attitude information is measured 
any event, inertial reference unit 34 provides instanta- 15 by the inertial reference unit 34 at the frame rate of 
neous position (x,y,z) and attitude (pitch, roll, yaw) data television camera 30. In processor 52, vehicle position 
from vehicle 22 to control station 12 via narrow band and attitude information is combined with camera 
RF link 16. Although this is real time data, that is a pointing information and formatted for transmission by 
frame rate of 30 Hz as will be described in more detail transmitter 54 and antenna 56. 

hereinafter, it requires only low data rates because only 20 The video output from television camera 30 is fed to 
six position and attitude numbers are needed to be trans- frame grabber 36 which samples the video at a reduced 
mitted every 30 milliseconds. The television camera frame rate. The image data in the sampled frame (i.e., 
video output can be in any convenient one of a number the snapshots) are further compressed in bandwidth 
of well known analog or digital formats. compression unit 38 utilizing standard bandwidth coiri- 

Other vehicle equipment is assumedi to be on the 25 pression techniques such as adaptive delta pulse code 
vehicle 22 such as servos 36 which actuate various parts modulation (ADPCM), discrete cosine transform or a 
of the driving gear on the vehicle upon receipt in proper hybrid technique to further reduce the bit rate to 16 to 
form of control signals from the operator 20 via com- 64 thousand bits per second. The compressed video is 
munications link 18. In addition, various operational supplied to transmitter unit 60 and antenna 62 for trans- 
data is fed back to control station 12 via this link. 30 mission to the control station 12. 

Unique to the requirements of this invention is the At control station 12, as shown in FIG. 3, receiving 
single frame sampler 36 which samples the video from antenna 64 provides signals to narrow band data re- 
television camera 30 at a low rate of, for example, one ceiver 66. A reconstruction module 68, which is the 
frame every one to three seconds as contrasted with the complement of the compression processor 38 on the 
standard 30 frames per second rate, providing snapshots 35 vehicle 22, is connected to receiver 66. Module 68 re- ^ 
rather than a continuous video output. In computer constructs the single frames from the received corn- 
graphics systems, the sampler 36 is referred to as a pressed video data by employing a bandwidth recon- 
frame "grabber". There are many such frame grabbers struction technique which is the complement of the 
available commercially. The snapshots from the single compression technique employed by bandwidth corn- 
frame sampler 36 are further subjected to a conven- 40 pression unit 38 on the vehicle. The reconstructed sin- 
tional bandwidth reduction techniques by single frame gle frames are fed to frame memory 70 which performs 
compressor 38 to provide a total combined bandwidth multiple buffering so that a new frame can be accepted 
reduction in excess of 1000:1. from reconstruction module 68 while the previous 

This reduced bandwidth image data is transmitted via frame is being read out to a pipeline processor generally 
narrow band RF data link 14 to frame buffer 40 at the 45 indicated by reference numeral 80, which will be de- 
control station 12. Frame buffer 40 performs double scribed in more detail hereinafter, 
buffering so that a new snapshot image can be received The position and attitude data transmitted from the 
as the previously received frame is being fed to real time vehicle 22 is also received at the control station 12. The 
transform unit 42. * Instantaneous vehicle position and same receiving equipment may be used at the control 
attitude information is continuously transmitted via the 50 station for both the video data and the position and 
same or a separate narrow band data link 16 to trans- attitude data. However, for this description, a separate 
form computer 44 which uses this data to compute the receiving antenna 72 and data receiving unit 74 are 
coefficients (parameters) necessary for transforming the shown. Antenna 72 and receiver 74 may also be the 
single frame image, or snapshot, into real time video receiving end of a transceiver or part of a communica- 
images. The transform coefficients (parameters) are 55 tion interface already provided in a robotic vehicle to 
supplied to real time perspective transform unit 42 feed back information on vehicle position and other 
which performs the actual transformation. The trans- operating conditions, such as for example odometer, 
form computer 44 and real time perspective transform speedometer and other such gauge readings, 
unit 42 comprise a pipeline processor that computes in The position and attitude data are fed from receiver 
real time perspective images from the snapshots of 60 74 to a control processor 76 which, in addition to inter- 
scene images of the vehicle's operating environment. facing with the receiver, transforms the data into the 
Pipeline processors and perspective scene image gener- parameters required by a pipeline controller 82 in pipe- 
ation are discussed in more detail in U.S. Pat. Nos. line processor 80 to perform the perspective transfor- 
4,645,459 to Graf et al. and 4,667,190 to Fant and U.S. mation into real time images of the single frame data 
patent application Ser. No. 06/788,652 to Fant men- 65 from memory 70. Control processor 76 thus outputs to 
tioned above. The real time images, including perspec- pipeline controller 82 the control points of the transfor- 
tive changes, are presented to operator 20 at display 48. mation which the pipeline controller needs to compute 
This display presents images that have the appearance the scan line warp coefficients required by the pipeline 
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processor 80. Control processor 76 also controls frame 
rates for transformation and sequences the pipeline con- 
troller 82 to move the frame through the successive 
stages of the pipeline process. 

The pipeline controller 82, as is typical, provides 5 
clock signals for input of the image frames from frame 
memory 70 to the pipeline processor 80 as well as read- 
ing them out of memories associated with the various 
stages of the pipeline processor. It also computes the 
scan line "warp* 1 coefficients on a scan line by scan line 10 
basis to implement the entire algorithm to provide the 
transformation from frame data to real time images 
having the desired perspective. To implement the trans- 
formation, the pipeline processor comprises an object 
memory 84 feeding a vertical processor 86, which is a IS 
pixel processor, in turn feeding a rotator memory 88 
which performs the scan line conversion. The output 
from the rotator memory 88 is read out to a horizontal 
processor 90, which is a pixel processor. 

The output of pipeline processor 80 is frame-by-frame 20 
image data transformed to real time and is in digital 
form. Video output unit 92 performs a digital-to-analog 
conversion and reinserts the requisite synchronizing 
and blanking control signals so that the real time video 
image data can be presented on a conventional CRT 25 
display 94. The information thus provided is used by 
operator 20 located at control station 12 to drive and 
otherwise control remote vehicle 22. Of course, many 
variations may be used in the bandwidth compression, 
reconstruction and transformation of the frame or snap- 30 
shot image data within the scope of the invention as 
well as the communication of the data between the 
vehicle 22 and the control station 12. 

FIG. 4 shows the general functional blocks of the 
pipeline warp -process. The function of the warp proces- 35 
sor is to (1) reconstruct the real-time video from the 
compressed video data that has been transmitted across 
the data link to the image warp processor, and (2) warp 
this image into the correct perspective based upon the 
current inertial data. The pipeline is capable of perform- 40 
ing any arbitrary perspective transformation. The data 
comes into the frame buffer or object memory 84. A 
frame microprocessor 96, which is part of the pipeline 
controller 82, receives warp image transform parame- 
ters from control processor 76 and supplies a memory 45 
mode control to the frame buffer 84, and a line micro- 
processor 98, also part of the pipeline controller 82, 
supplies an address to the frame buffer 84 in response to 
initial fust pass values and deltas (i.e., changes in values) 
from microprocessor 96. Data from the frame buffer 84 50 
is read out to a vertical processor, which is the pixel 
processor 86. The pixel processor 86 operates under the 
control of the line microprocessor 98 which provides a 
size factor input. The resulting data from the pixel pro- 
cessor 86 is alternately written into one or the other of 55 
frame buffers 88a or 886, which constitute the rotator 
memory 88. The addresses in the frame buffers 88a and 
886, where the data are written, are generated by the 
line microprocessor 98, and the frame microprocessor 
96 also provides a memory mode control to the frame 60 
buffers 88a and 886. As data is being written into one of 
the frame buffers 88a or 886 under the control of line 
microprocessor 98, data is being read out of one of the 
frame buffers 886 or 88a under the control of line micro- 
processor 100, also part of the pipeline controller 82. 65 
Line microprocessor 100 receives initial second pass 
values and deltas from the frame microprocessor 96 and 
generates addresses for the frame buffers 88a and 886 
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and a size factor input for a horizontal processor, which 
is the pixel processor 90. 

As may be appreciated from the foregoing descrip- 
tion, the control and processing inside the warp pipeline 
processor naturally partitions into three hierarchical 
stages, namely, the frame, line and pixel stages. The 
frame and line processing is low bandwidth than pixel 
processing and can be accomplished with available 
16-bit microprocessors, such as the Motorola 68000. 
The pixel processing is very high data rate and requires 
custom designed hardware. This hardware is described 
in detail in U.S. Pat. No. 4,645,459 to Graf et al. t for 
example, to which the reader is referred for further 
details. 

The frame level control involves setting up the mem- 
ories, communicating with the control processor 76 and 
initiating the line processors. Several computation tasks 
are carried out at the frame level. On the first pass, the 
vertical orientation is. determined as indicated by the 
intermediate image shown in FIG. 5. On the second 
pass, the horizontal orientation of this intermediate 
image is determined to generate the resulting image on 
the video screen. The warp image processor continu- 
ously warps the last frame received as a function of the 
difference between the vehicle's current location (deter- 
mined from navigational data transmitted to the control 
station 12 from the vehicle 10) and the vehicle's location 
when the image was acquired. The algorithm that is 
implemented is as follows: 

1) Check if a new image has arrived (images arrive 
every three seconds), If an entire new image has ar- 
rived, go to 2, otherwise go to 3. 

2) Establish the inertial coordinates of four control 
points of the image in order to define the warp mapping 
for this and subsequent frames (image plane locations 
for these points may be selected manually by the opera- 
tor or automatically). 

3) Derive the warp mapping as . a function of the 
vehicle's current location. 

4) Warp the image. 

The control or warp image processor 76, shown in 
FIG. 3, sends its control parameters to the pipeline 
controller 82. The pipeline controller 82 requires as 
input the control points of the transformation which is 
used to computer the scan line warp coefficients re- 
quired by the pipeline architecture. The primary func- 
tion of the pipeline controller 82 is to implement the 
warp algorithm which defines the perspective transfor- 
mation of a planar patch of image data. The patch to be 
transformed is rectangular and is specified by the coor- 
dinates of the four comers of the patch in the two-di- 
mensional image plane. The locations of the patch in 
three dimensions (after perspective transformation) is 
specified by the coordinates of the corners of the pro- 
jected image of the patch on the two-dimensional im- 
age. These two sets of coordinates for the four corners 
of the patch define the perspective transformation 
uniquely. 

As applied to the subject invention, the four corner 
locations for the planar patch before transformation 
could be the four corners of the image. Thus, the four 
coordinates for the four corners of the planar patch are 
obtained by calculating where rays from each of the 
four corners to the lens center intersect the image plane. 
A more appropriate set of corner points may be those 
points which cause minimal distortion in the warped 
imagery for nominal driving conditions of the vehicle. 
These corner or control point locations can be identi- 
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Tied by subjective evaluation of the image quality for 
various choices for the control point locations. 

The mapping performed by the warping algorithm 
defines the new image plane locations for the control 
points exactly. It is only approximately correct for the 5 
remaining points of the image plane. However, if the 
control points are chosen intelligently, the mapping will 
be nearly exact for the true corners of the fixed image. 
Therefore, the coordinate locations that define the warp 
transformation can be the original corners of the fixed 10 
image and the approximate locations of these corners, 
obtained by the bi-affine transformation. 

An alternative embodiment of the invention uses 
vehicle position and attitude information in a manner 
different from the transformation described above. In- 15 
stead of warping single frame images, they are pres- 
ented in frozen frame form. The vehicle's position and 
attitude information when the frozen frame image was 
made and its instantaneous position and attitude are 
both available at the control station. This information is 20 
used to construct a dynamic graphical overlay of the 
vehicle which is superimposed on the frozen frame 
presented to the operator. A display is provided in 
which the vehicle appears to move into the still frame, 
representing instantaneous vehicle position and en- 25 
abling the operator to control the vehicle. The next still 
frame received replaces the old still frame and the cur- 
rent position of the vehicle is represented relative to the 
new frame being displayed. Thus, although the entire 
viewpoint of frozen frames changes, the relative posi- 30 
tion of the vehicle does not, and continuous operator 
control is possible. The implementation of this embodi- 
ment is possible using known three-dimensional graphic 
systems capable of rendering frame models at real time 
and near real , time rates. However, this form of the 35 
invention does not use a conventional continuous video 
display at the man-machine interface and may require 
the operator to perform additional tasks. 

While the invention has been described in terms of a 
preferred embodiment, those skilled in the art will rec- 40 
ognize that the invention can be practiced with modifi- 
cation within the spirit and scope of the appended 
claims. 

Having thus described our invention, what we claim 
as new and desire to secure by Letters Patent is as fol- 45 
lows: 

1. A system for driving from a remote control station 
a vehicle comprising; 

a sensor on the vehicle for providing snapshot images 
from the vehicle of an environment in which the 50 
vehicle is operating; 

motion detecting means on the vehicle for providing 
information describing the motion of the vehicle; 

transmitting means on the vehicle for transmitting the 
snapshot images and the information describing the 55 
motion of the vehicle to the remote control station; 
and 

image generation means at the remote control station 
for combining the snapshot images and the infor- 
mation describing the motion of the vehicle into a 60 
display that can be utilized for driving the vehicle 
from the remote control station. 

2. The system as recited in claim 1 wherein the mo- 
tion detecting means includes an inertial reference unit 
supplying vehicle position and attitude data. 65 

3. The system as recited in claim 2 wherein the sensor 
is a television camera mounted on the vehicle further 
comprising a pan and tilt mechanism on which the cam- 
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era is mounted, the pan and tilt mechanism being refer- 
enced to said inertial reference unit for providing in- 
stantaneous position and attitude data. 

4. The system recited in claim 1 wherein the transmit- 
ting means includes a narrow band communications 
link. 

5. The system as recited in claim 4 wherein the sensor 
includes a television camera providing a video output. 

6. The system as recited in claim 5 wherein the trans- 
mitting means includes data compression means for 
compressing the video output for transmission by the 
narrow band communication link. 

7. The system as recited in claim 6 wherein the data 
compression means includes sampling means for sam- 
pling the video output to provide frame-by-frame im- 
ages for transmission. 

8. The system as recited in claim 7 wherein the data 
compression means includes means for compressing the 
video data in the frame images. 

9. A system for driving a vehicle from a remote con- 
trol station comprising: 

a television sensor on the vehicle for providing video 
images from the vehicle of an environment in 
which the vehicle is operating; 

sampling means on the vehicle for sampling the video 
images at a reduced frame rate, thereby providing 
video image data at a reduced bandwidth; 

data compression means on the vehicle for further 
reducing the bandwidth of the video image data; 

inertial means on the vehicle for providing instanta- 
neous vehicle position and attitude data at a low 
data rate for transmission to the control station; 

a narrow band radio frequency data link for transmit- 
ting the reduced bandwidth video data and low 
data rate position and attitude data to the control 
station; 

receiving means at the control station for receiving 
the reduced bandwidth video data and low data 
rate position and attitude data transmitted from the 
vehicle; 

decompression means at the control station for repro- 
ducing the video images at the reduced frame rate; 

memory means at the control station for temporarily 
storing the reproduced video, images; 

coefficient generation means at the control station for 
receiving the low data rate position and attitude 
data and computing the coefficients required to 
transform images in said memory means into real 
time video images; 

image generation means responsive to said coeffici- 
ents for transforming images in said memory means 
into real time video images; and 

display means for displaying the real time video im- 
ages to assist an operator in remotely controlling 
the vehicle. 

10. A method for driving a vehicle from a remote 
control station comprising the steps of: 

sensing images from the vehicle at a low frame rate to 
produce a series of snapshots of an environment in 
which the vehicle is operating; 

determining the instantaneous position and attitude of 
the vehicle; 

transmitting the snapshots and position and attitude 
to the remote control station; 

temporarily storing the snapshots at the remote con- 
trol station and extrapolating between snapshots 
based on the vehicle position and attitude to gener- 
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ate a continuous real time image of the vehicle 

environment; and 
displaying the continuous real time image to assist an 5 

operator in controlling the vehicle. 
11. A method for transmitting and receiving video 
data in a television format comprising the steps of: 

10 



14 

sampling video data from a camera at a reduced 
frame rate; 

sensing instantaneous position and attitude informa- 
tion of the camera; 

using the instantaneous position and attitude informa- 
tion to extrapolate between sampled video data to 
generate real time video image data; and 

displaying the generate real time video image data. 

* * » * * • 
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